Add Conditional Value at Risk (CVAR) metric#100
Add Conditional Value at Risk (CVAR) metric#100abdelrahman-ayad wants to merge 23 commits intomainfrom
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #100 +/- ##
==========================================
+ Coverage 83.13% 83.57% +0.43%
==========================================
Files 45 45
Lines 2325 2466 +141
==========================================
+ Hits 1933 2061 +128
- Misses 392 405 +13 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…capacity calculations
There was a problem hiding this comment.
Pull request overview
Adds a Conditional Value at Risk (CVAR) (and normalized NCVAR) reliability metric for tail-risk assessment of shortfalls, including capacity-shortfall variants, and updates simulation recording + tests accordingly.
Changes:
- Introduces
CVAR/NCVARmetric types (plus display/validation) and computes them forShortfallResultandShortfallSamplesResult. - Extends result recording/finalization to retain per-sample unserved energy and max capacity shortfall needed for CVAR.
- Adds/updates unit tests and reference values (including system test data) to cover CVAR/NCVAR behavior.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| PRASCore.jl/src/Results/metrics.jl | Adds CVAR/NCVAR metric structs and show methods; adjusts MeanEstimate for singleton samples. |
| PRASCore.jl/src/Results/Shortfall.jl | Stores per-sample unserved energy and adds CVAR/NCVAR computations for ShortfallResult. |
| PRASCore.jl/src/Results/ShortfallSamples.jl | Adds max capacity shortfall storage and CVAR/NCVAR computations for ShortfallSamplesResult. |
| PRASCore.jl/src/Results/Results.jl | Exports CVAR/NCVAR and adds broadcast convenience methods. |
| PRASCore.jl/src/Simulations/recording.jl | Records per-sample UE totals into the shortfall accumulator. |
| PRASCore.jl/src/Systems/TestData.jl | Adds expected CVAR reference values for simulation tests. |
| PRASCore.jl/test/Results/metrics.jl | Adds tests for CVAR/NCVAR formatting and domain checks. |
| PRASCore.jl/test/Results/shortfall.jl | Adds correctness tests for CVAR/NCVAR on results (overall/region/period). |
| PRASCore.jl/test/Simulations/runtests.jl | Adds simulation-level assertions for CVAR values. |
| PRASCore.jl/test/dummydata.jl | Adds dummy alpha/sample vectors for new result fields. |
| PRAS.jl/test/runtests.jl | Expands integration tests to validate CVAR/NCVAR return types for both result kinds. |
Comments suppressed due to low confidence (1)
PRASCore.jl/test/Results/shortfall.jl:43
- Typo in variable name:
cap_shortfalis missing anl(should becap_shortfall) to match intent and improve readability.
cap_shortfal = vec(reshape(result.capacity_shortfall_mean, 1, :))
var = quantile(cap_shortfal, alpha)
tail_losses = cap_shortfal[cap_shortfal .>= var]
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| nsamples = first(acc.unservedload_total.stats).n | ||
|
|
||
| p2e = conversionfactor(L,T,P,E) | ||
| capacity_shortfall_mean = ue_regionperiod_mean |
There was a problem hiding this comment.
capacity_shortfall_mean = ue_regionperiod_mean aliases the same matrix; the subsequent in-place scaling (.*=) will also scale capacity_shortfall_mean, so it will no longer be in capacity units. Make capacity_shortfall_mean a copy (or avoid in-place scaling) before converting ue_regionperiod_mean.
| capacity_shortfall_mean = ue_regionperiod_mean | |
| capacity_shortfall_mean = copy(ue_regionperiod_mean) |
|
|
||
| estimate = x[] | ||
| var = quantile(estimate, alpha) | ||
| tail_losses = estimate[estimate .> var] |
There was a problem hiding this comment.
This overload uses a strict > comparison for tail selection, while the other CVAR overloads (and tests) use .>=. This can drop observations equal to the quantile threshold and makes behavior inconsistent across overloads. Use the same inclusion rule consistently (typically .>= var).
| tail_losses = estimate[estimate .> var] | |
| tail_losses = estimate[estimate .>= var] |
| function NCVAR(x::ShortfallSamplesResult{N,L,T,P}, cvar::CVAR) where {N,L,T,P} | ||
| demand = sum(x.regions.load) | ||
|
|
||
| if demand > 0 | ||
| ncvar = div(cvar.cvar, demand/1e6) | ||
| var = div(cvar.var, demand/1e6) | ||
| else | ||
| ncvar = MeanEstimate(0.) | ||
| var = MeanEstimate(0.) | ||
| end | ||
|
|
||
| return NCVAR(ncvar, cvar.alpha, var) |
There was a problem hiding this comment.
Two issues: (1) NCVAR stores var::Float64, but the else branch sets var = MeanEstimate(0.), which will fail construction/type expectations. (2) div on floating-point inputs truncates (and may not be defined for MeanEstimate), which is not appropriate for normalization. Use true division for scaling, and ensure the var argument passed to NCVAR(...) is always a Float64.
| function NCVAR(x::ShortfallResult{N,L,T,E}, cvar::CVAR) where {N,L,T,E} | ||
| demand = sum(x.regions.load) | ||
|
|
||
| if demand > 0 | ||
| ncvar = div(cvar.cvar, demand/1e6) | ||
| var = div(cvar.var, demand/1e6) | ||
| else | ||
| ncvar = MeanEstimate(0.) | ||
| var = MeanEstimate(0.) | ||
| end | ||
|
|
||
| return NCVAR(ncvar, cvar.alpha, var) |
There was a problem hiding this comment.
Same normalization problems as in ShortfallSamples.jl: div will truncate for floats (and likely won’t work for MeanEstimate), and the else branch passes a MeanEstimate where NCVAR expects var::Float64. Switch to non-truncating division and keep var as Float64 on all paths.
| CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, t::ZonedDateTime) = | ||
| CVAR.(x, alpha, x.regions.names, t) | ||
|
|
||
| CVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) = | ||
| CVAR.(x, alpha, r, x.timestamps) | ||
|
|
||
| CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) = | ||
| CVAR.(x, alpha, x.regions.names, permutedims(x.timestamps)) | ||
|
|
||
| NCVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) = | ||
| NCVAR.(x, alpha, r, x.timestamps) | ||
|
|
||
| NCVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) = | ||
| NCVAR.(x, alpha, x.regions.names, permutedims(x.timestamps)) |
There was a problem hiding this comment.
These broadcast convenience methods call NCVAR.(x, alpha, ...), but in this PR the implemented NCVAR overloads take a cvar::CVAR metric (e.g., NCVAR(x, cvar) / NCVAR(x, cvar, r)), not (x, alpha, ...). As written, these definitions will raise MethodError unless additional (x, alpha, ...) overloads exist. Prefer updating these convenience methods to accept cvar::CVAR (or implement the corresponding NCVAR(x, alpha, ...) methods consistently).
| CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, t::ZonedDateTime) = | |
| CVAR.(x, alpha, x.regions.names, t) | |
| CVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) = | |
| CVAR.(x, alpha, r, x.timestamps) | |
| CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) = | |
| CVAR.(x, alpha, x.regions.names, permutedims(x.timestamps)) | |
| NCVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) = | |
| NCVAR.(x, alpha, r, x.timestamps) | |
| NCVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) = | |
| NCVAR.(x, alpha, x.regions.names, permutedims(x.timestamps)) | |
| CVAR(x::AbstractShortfallResult, cvar::CVAR, ::Colon, t::ZonedDateTime) = | |
| CVAR.(x, cvar, x.regions.names, t) | |
| CVAR(x::AbstractShortfallResult, cvar::CVAR, r::AbstractString, ::Colon) = | |
| CVAR.(x, cvar, r, x.timestamps) | |
| CVAR(x::AbstractShortfallResult, cvar::CVAR, ::Colon, ::Colon) = | |
| CVAR.(x, cvar, x.regions.names, permutedims(x.timestamps)) | |
| NCVAR(x::AbstractShortfallResult, cvar::CVAR, r::AbstractString, ::Colon) = | |
| NCVAR.(x, cvar, r, x.timestamps) | |
| NCVAR(x::AbstractShortfallResult, cvar::CVAR, ::Colon, ::Colon) = | |
| NCVAR.(x, cvar, x.regions.names, permutedims(x.timestamps)) |
| val(cvar) >= 0 || throw(DomainError( | ||
| "$val is not a valid CVAR")) |
There was a problem hiding this comment.
The error message interpolates $val, which is not the CVAR value and will render as the val function rather than the offending number. Interpolate the actual invalid value (e.g., val(cvar)), and consider including which field failed (e.g., cvar vs capacity_cvar).
|
|
||
| function NCVAR(ncvar::MeanEstimate, alpha::Float64, var::Float64) | ||
| val(ncvar) >= 0 || throw(DomainError( | ||
| "$val is not a valid NCVAR")) |
There was a problem hiding this comment.
Same as CVAR: this interpolates $val (the function), not the numeric value that violated the domain constraint. Interpolate the invalid estimate (e.g., val(ncvar)) so the message is actionable.
| "$val is not a valid NCVAR")) | |
| "$(val(ncvar)) is not a valid NCVAR")) |
2 participants
This PR adds the conditional value at risk (CVAR) metric to assess tail risk shortfalls. CVAR measures the expected value observations above a pre-determined threshold$\alpha$ (Nth percentile).
The current CVAR implementation measures two risk metrics: total unserved energy and capacity shortfalls, as follow:
ShortfallandShortfallSamples. To calculate the values from theShortfall, an additional vector of total unserved energy for each sample is stored before being removed from memory.CVARmetric reports the unserved energy by default (1), and it has capacity cvar fields for (2) if calculated from theShortfallSamplesor (3) if calculated from theShortfallAttached are
@benchmarktesting forShortfallandShortfallSamplescomparison after implementing the CVAR metrics.Main branch: